A Hybrid Approach for Refreshing Web Page Repositories

Web pages change frequently and thus crawlers have to download them often. Various policies have been proposed for refreshing local copies of web pages. In this paper, we introduce a new sampling method that excels over other change detection methods in experiments. Change Frequency (CF) is a method that predicts the update frequency of the pages and achieves a better efficiency in comparison with the sampling method, in the long run. Here, we suggest an improvement to CF. We also propose a new hybrid method that is a combination of our new sampling and CF. We show how the new method succeeds in preserving the benefits of both methods while reducing their fluctuations. We then extend our algorithms to handle a more realistic condition when our limited resource is crawler’s bandwidth, not its number of pages. Our experimental results on more than two million crawled pages, demonstrate the efficiency of our algorithms keywords: web crawling, change detection, sampling, change frequency, hybrid algorithms